Image processing apparatus and method for previewing still and motion images

ABSTRACT

An image processing apparatus and method for regulating viewing image output of the image processor which processes motion and still images in a time divisional manner are provided. The image processing apparatus includes an image processor processing the viewing and still images in a time divisional manner, a timing manager storing the viewing image processed by the image processor temporarily and outputting the viewing image at a preset time of a frame period, and a controller controlling, in response to a capture request, the image processor to process the still image at a residual part of a current frame to be at a beginning part of a next frame and to process the buffered viewing image to be at the residual part of the next frame, wherein the residual part of the next frame is a part which remains after processing the still image completely.

PRIORITY

This application claims the benefit under 35 U.S.C. §119(a) of a Koreanpatent application filed on May 3, 2012 in the Korean IntellectualProperty Office and assigned Serial No. 10-2012-0046617, the entiredisclosure of which is hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus andmethod of a camera. More particularly, the present invention relates toan apparatus and method for regulating viewing of an image output froman image processor which processes motion and still images in a timedivisional manner.

2. Description of the Related Art

A camera device and camera-equipped electronic device are capable ofprocessing high quality images and providing various user conveniencefunctions. The related-art camera device is equipped with an imagesensor or camera sensor capable of processing full High Definition (HD)or higher resolution images.

The camera device displays the image sensed by the camera sensor in apreview mode and saves the image acquired by the camera sensor inresponse to the push on a shutter button. A shutter delay or shutter lagbetween triggering the shutter and when the camera records a photographmay exist. For example, there is the time difference between when theuser presses the shutter and when the image processor processes thephoto completely, and this is referred to as a lag. The shutter delay orshutter lag is the reason why a user may not capture an intended shot.

Also, the related-art camera device is limited in processing the previewand still images simultaneously or the motion and still imagessimultaneously. In the case of processing motion and still imagessimultaneously, the limited processing capability of the image processorof the related-art camera device is likely to fail processing the stillimage to guarantee the motion image output at the output frame rateequal to the input frame rate.

The above information is presented as background information only toassist with an understanding of the present disclosure. No determinationhas been made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the present invention.

SUMMARY OF THE INVENTION

Aspects of the present invention are to address at least theabove-mentioned problems and/or disadvantages and to provide at leastthe advantages described below. Accordingly, an aspect of the presentinvention is to provide an apparatus and method for outputting theprocessed motion image at a predetermined frame rate in a camera deviceor camera-equipped electronic device. Here, the motion image can be apreview image and/or an on-recording motion image, and the preset timeof a frame period can be a time delayed by as much as a predeterminedduration from the frame end time or frame start time.

In the case of processing the still and motion images simultaneously,the image processor processes and buffers the motion image first,processes the still image subsequently, and then outputs the processedmotion image at the time of a frame period. If a frame image forprocessing the still image is received in the next frame duration, thenthe image processor buffers the still image, then processes the motionimage of the next frame after processing the still image, and thenoutputs the processed motion image at the preset time of the frameperiod.

In accordance with an aspect of the present invention, an imageprocessing apparatus is provided. The apparatus includes an imageprocessor including an image scaler scaling an image generated by acamera into a viewing image at every frame period, a viewing imagebuffer buffering the viewing image, and a still image buffer bufferingthe image generated by the camera as a still image, the image processorprocessing the viewing and still images in a time divisional manner, atiming manager storing the viewing image processed by the imageprocessor temporarily and outputting the viewing image at a preset timeof a frame period, and a controller controlling, in response to acapture request, the image processor to process the still image at aresidual part of a current frame to be at a beginning part of a nextframe and to process the buffered viewing image to be at the residualpart of the next frame, wherein the residual part of the next frame is apart which remains after processing the still image completely.

In accordance with another aspect of the present invention, an imageprocessing method is provided. The method includes processing an imagegenerated by the camera at every frame period by converting the imagefrom the camera to a still image and a viewing image, buffering thestill and viewing images, and processing the buffered viewing image,storing the processed viewing image temporarily; and outputting thetemporarily stored viewing image at a preset time of a frame period.

Other aspects, advantages, and salient features of the invention willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses exemplary embodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainexemplary embodiments of the present invention will be more apparentfrom the following description taken in conjunction with theaccompanying drawings, in which:

FIG. 1 is a block diagram illustrating a configuration of a cameradevice according to an exemplary embodiment of the present invention;

FIGS. 2A and 2B are block diagrams illustrating exemplary configurationsof an image processor, such as the image processor of FIG. 1, accordingto an exemplary embodiment of the present invention;

FIGS. 3A and 3B are block diagrams illustrating other exemplaryconfigurations of an image processor, such as the image processor ofFIG. 1, according to an exemplary embodiment of the present invention;

FIG. 4 is a block diagram illustrating an image processor processingmotion and still images according to an exemplary embodiment of thepresent invention;

FIG. 5 is a timing diagram illustrating operation timings of an imageprocessor, such as the image processor of FIG. 4, according to anexemplary embodiment of the present invention;

FIG. 6 is a timing diagram illustrating operation timings of an imageprocessor, such as the image processor of FIG. 4, according to anexemplary embodiment of the present invention;

FIG. 7 is a timing diagram illustrating operation timings of an imageprocessor, such as the image processor of FIG. 4, according to anexemplary embodiment of the present invention;

FIG. 8 is a timing diagram illustrating operation timings of an imageprocessor, such as the image processor of FIG. 4, according to anexemplary embodiment of the present invention;

FIG. 9 is a flowchart illustrating an image processing method of acamera device or camera-equipped electronic device according to anexemplary embodiment of the present invention;

FIG. 10 is a flowchart illustrating a viewing image processing procedurein a motion image processing mode of an image processor according to anexemplary embodiment of the present invention;

FIG. 11 is a flowchart illustrating a procedure of processing motion andstill images in an image processing method according to an exemplaryembodiment of the present invention;

FIG. 12 is a flowchart illustrating the procedure of controlling viewingimage output timing of an image processor according to an exemplaryembodiment of the present invention;

FIG. 13 is a flowchart illustrating a procedure of outputting a viewingimage in processing motion image in a motion image processor accordingto an exemplary embodiment of the present invention; and

FIG. 14 is a flowchart illustrating the procedure of outputting aviewing image when an image processor processes a motion and stillimages in a time divisional manner according to an exemplary embodimentof the present invention.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of exemplaryembodiments of the invention as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the embodiments described hereincan be made without departing from the scope and spirit of theinvention. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of theinvention. Accordingly, it should be apparent to those skilled in theart that the following description of exemplary embodiments of thepresent invention is provided for illustration purpose only and not forthe purpose of limiting the invention as defined by the appended claimsand their equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

An image processor of a camera device or a camera-equipped electronicdevice according to an exemplary embodiment of the present inventionprocesses every frame image output by the camera in a camera operationmode, e.g., a preview mode to generate an image as preview or motionimage, hereinafter, referred to as a motion image or a viewing image,and another image to be stored as a still image, hereinafter, referredto as a capture image. Here, the viewing image has a size equal to orless than that of the capture image. In this case, the camera deviceacquires an image from the camera at every frame and converts theacquired image to the viewing image being displayed on the display aspreview image while buffering the camera's full resolution image to bestored as still image. If a capture request is input in this state, thenthe camera device selects a frame image at the time when the capturerequest is detected, which is a frame image with zero shutter lag, amongthe buffered frame images and encodes the selected frame imagecompressively, the encoded image being stored as the capture image.

Typically, the camera device has a shutter lag between when the userpresses the shutter and when the camera captures the image. The cameradevice, according to an exemplary embodiment of the present invention,is provided with a buffer for storing the camera's full resolutionimages in separation with the viewing image and buffers the fullresolution images while the camera is operating. The camera deviceselects and processes one of the buffered frame images at shootingrequest timing so as to achieve zero shutter lag. In an exemplaryembodiment of the present invention, the buffer can be configured tostore a predetermined number of frame images and large enough tocompensate for the shutter lag in size. Typically, the shutter lag maybe a time length of about 2 frames period. Accordingly, it is preferredto configure the buffering window to be less than 5 frames.

The image processor according to an exemplary embodiment of the presentinvention processes the viewing and still images in a time divisionalmanner. In the motion image processing mode, e.g., preview mode ormotion image recording mode, the image processor establishes a motionimage processing path. In the capture mode, the image processorestablishes a viewing image processing path first to process the motionimage and then a still image processing path to process the still imageduring the residual part of the frame. At this time, the still image canbe acquired as a high pixel image with an advanced image sensor of thecamera.

In the case that the camera outputs the high pixel image, the imageprocessor may not process both the motion and still images in one frameperiod. In order to overcome this problem, the image processor accordingto an exemplary embodiment of the present invention buffers the viewingimages processed in motion image processing mode and then outputs duringa preset time of the frame period, e.g., a frame end or a time close tothe frame end. This is to match the current viewing image output time tothe previous frame viewing image output time even in the case that thestill images in two consecutive frames are processed such that theviewing image of the next frame is processed after processing the stillimage. For example, the image processor, according to an exemplaryembodiment of the present invention, processes, in the case ofprocessing both the motion and still images simultaneously, the motionimage first in the frame period, the still image during the residualpart of the frame and at the beginning of the next frame continuously,and then the motion image in the residual part of the next frame period.

For example, the image processor, according to an exemplary embodimentof the present invention, processes, in the frame period in which themotion and still images have to be processed, the motion image first atthe beginning of the frame, the still image at the residual part of theframe and the beginning part of the next frame, and the motion image atthe residual part of the next frame after completion of processing thestill image.

In the following description, a term ‘full resolution image’ denotes theimage output by the camera in a state of being not scaled. A term‘viewing image’ denotes the image displaying on the display unit in apreview mode or capable of being stored as a motion image in a state ofbeing scaled to a predetermined size or resolution. The preview and themotion images derived from the viewing image may have the same sizeresolution. A term ‘capture image’ denotes the image capable of beingstored as a still image and can be a camera's full resolution image oran image scaled to a predetermined size. In an exemplary embodiment ofthe present invention, the description is made under the assumption thatthe capture image is the camera's full resolution image.

A term ‘image scaling’ denotes adjusting the full resolution image to apredetermined size or resolution. In an exemplary embodiment of thepresent invention, image scaling can be implemented through resizingprocess and/or summing and averaging process. Here, resizing is a methodfor adjusting image size through decimation, interpolation, and/or crop.The summing and averaging process is the method for adjusting the numberof pixels by rendering adjacent pixels into one pixel and may befollowed by decimation, interpolation, and/or crop.

FIG. 1 is a block diagram illustrating a configuration of a cameradevice according to an exemplary embodiment of the present invention.

Referring to FIG. 1, a camera 110 takes an image by means of an internalsensor in a camera-driving mode. The camera 110 includes an optical unit(not shown), an image sensor (not shown), and a signal processor (notshown). The optical unit is driven by a Mecha-shutter (not shown), amotor (not shown), and an actuator (not shown) and, particularly,performs zooming and focusing operations with the actuator. The opticalunit takes an image, and the image sensor converts the image taken bythe optical unit to an electrical signal. Here, the image sensor can bea high resolution image sensor such as Complementary Metal OxideSemiconductor (CMOS) or Charge Coupled Device (CCD) sensor. The imagesensor may include a global shutter. The global shutter is capable ofperforming a function similar to that of the Mecha-shutter embedded inthe sensor.

In an exemplary embodiment of the present invention, the image sensorcan be a sensor capable of sensing Ultra High Definition (UHD) or higherresolution image. The image sensed by the image sensor is converted intoa digital image by the signal processor. The data output by the camera110 can be Bayer data, raw data, or any other similar and/or suitabletype of data.

A control unit 100 controls overall operations of the camera device.Particularly in an exemplary embodiment of the present invention, thecontrol unit 100 controls an image processor 200 to process the viewingand capture images in a time divisional manner. The control unit 100controls the image processor 200 to processes the viewing image in thepreview mode and the viewing image for display and the capture image forrecording in the capture mode.

The image processor 200 converts the image taken by the camera 110 to aviewing image in the preview mode and in the motion image recordingmode. The image processor 200 processes the viewing image at thebeginning of the current frame period, the still image at the residualpart of the current frame and the beginning of the next frame, and theviewing image of the next frame after the completion of the still imageprocessing.

A timing manager 170 stores the viewing image processed by the imageprocessor 200 temporarily and outputs the temporarily stored viewingimage at the preset time of the frame period. The timing manager 170 iscapable of outputting the viewing image at a preset time of every frameperiod. Here, the timing manager 170 can be a buffer, and the presettime of the frame period can be the frame end part or a part close tothe frame end.

A still image codec 190 encodes and compresses the still image output bythe image processor 200. And the compressed image is stored a storageunit 120. Here, the still image codec can be a JPEG codec. The viewingimage can be used for storing a motion image. If a motion image shootingrequest is detected, a motion image codec 180 encodes the viewing imagesoutput by the image processor 200 into a motion image which is stored inthe storage unit 120. Here, the motion image can be the same size as theviewing image. The viewing image may have a number of pixels greaterthan that of the preview image. In such a case, the image processor 200may include a preview image scaler and a motion image scaler or a commonimage scaler capable of adjusting the scale ratio. If the motion imagerecording is requested, the control unit 100 controls the imageprocessor 200 to scale the image to the size of the motion viewing imagesuch that the image processor 200 scales the camera's full resolutionimage to the motion image size. The motion image encoding can beperformed with various motion image codecs, such as an H.264 codec orany other similar and/or suitable codec.

The storage unit 120 is a memory for storing the still and motion imagesacquired by processing the images. A display unit 130 displays theviewing images output as adjusted in display time by the timing manager170. The display unit 130 can be any Liquid Crystal Display (LCD), anOrganic Light Emitting Diode (OLED) display or any other similar and/orsuitable type of display device or unit. An input unit 140 is providedwith a plurality of keys capable of generating commands for configuringand executing various functions of the camera device. Here, the keys ofthe input unit 140 can be provided outside of the camera device in theform of buttons and virtual keys presented on a touch panel included inthe camera device. The display unit 130 and the input unit 140 can beintegrated into a touchscreen.

The control unit 100 controls the operations of the camera deviceaccording to the control command input through the input unit 140. If acamera driving command is input through the input unit 140, the controlunit 100 controls the camera 110 and the image processor 200 to operatein the preview mode. If a capture request is input through the inputunit 140, the control units selects the full resolution image taken atthe time when the capture request is input from a still image buffer 250and controls a multiplexer 270 and a demultiplexer 280 (see FIGS. 3A and3B) in a time divisional manner such that the viewing image is displayedon the screen of the display unit 130 while the selected full resolutionimage is encoded to be stored in the storage unit 120. In the previewmode or motion image recording mode, the control unit 100 controls theimage processor 200 to process the viewing image, the processed viewingimage being stored in the timing manager 170 temporarily. Afterward, thecontrol unit controls the timing manager 170 to output the temporarilystored viewing image in the preset time of the corresponding frameperiod. In the capture mode, the control unit 100 controls the imageprocessor 200 to process the viewing image at the beginning part of thecurrent frame and store the processed viewing image in the timingmanager 170 temporarily and controls the image processor 200 toprocesses the still image at the residual part of the current frame andat the beginning part of the next frame continuously. Once the stillimage processing has completed, the control unit 100 controls the imageprocessor 200 to process the viewing image of the next frame, theprocessed viewing image being stored in the timing manager 170temporarily, and controls the timing manager 170 to output the viewingimage at the preset time of the frame period.

The image processor according to an exemplary embodiment of the presentinvention controls to display the viewing image at a preset time ofevery frame period, e.g., at the end part and a part close to the end ofthe frame period so as to regulate the display operation even when thestill image processing time elongates.

FIGS. 2A, 2B, 3A, and 3B are block diagrams illustrating configurationsof an image processor, such as the image processor of FIG. 1, accordingto an exemplary embodiment of the present invention.

Referring to FIG. 2A, a pre-processor 210 pre-processes the imageacquired from the camera 110. The pre-processing function includes atleast one of Auto White Balance (AWB), Auto Exposure (AE), Auto Focusing(AF), extraction and processing, lens shading correction, dead pixelcorrection, knee correction, and other similar and/or suitablepre-processing functions.

An image scaler 220 is responsible for scaling the camera's fullresolution image to the viewing image size for display on the screen ofthe display unit 130 and/or the motion image size for the motion imagebeing recorded. A viewing image buffer 230 buffers the viewing imageoutput from the image scaler 220. Here, the viewing image buffer 230 isresponsible for correcting the time for processing the viewing imagewithin the frame period.

A still image buffer 250 buffers the camera's full resolution imageoutput from the pre-processor 210. Here, the still image buffer 250 isconfigured to a size large enough to store a number of frame imageswhich is capable of compensating for the shutter lag of the cameradevice, e.g., equal to or less than 5 frame images, in a ring bufferstructure. The still image buffer 250 buffers the full resolution imageoutput from the pre-processor 210 at every frame and, if requested, animage selected from the buffer under the control of the control unit100.

A post-processor 240 receives the viewing image and/or the still imageoutput by the viewing image buffer 230 and/or the still image buffer 150in a time divisional manner and performs color interpolation, noisereduction, color correction, and image conversion on an input image togenerate YUV data. The post-processor 240 may include a colorinterpolator (not shown), an Image Processing Chain (IPC) (not shown),and an image converter (not shown). The color interpolator isresponsible for color interpolation for converting the Bayer data to thecolor image. As described above, the image sensor of the camera 110 canbe a CCD or CMOS image sensor. The CCD/CMOS image sensor may use a colorfilter array, and each pixel sensor has one of three color channels forgenerating a color image. The color interpolator is responsible forconverting the pixels of the image output from the camera 110 to thecolor RGB as three colors of the pixels, which may be a full colorconversion. The color interpolator is responsible for colorinterpolation function based on the correlation among the adjacentpixels. Basically, the process before the color interpolation isreferred to as pre-process and the process after the color interpolationis referred to as post-process. The IPC performs noise reduction, gammacorrection, and luminance correction. The image converter convertspost-processed raw data or Bayer data to YUV data. For example, thepost-processor 240 performs color interpolation on the pre-processedimage and converts the post-processed image to a YUV image.

A parser 260 parses the post-processed image to the viewing and/or stillimages. The parser 260 outputs the image from the post-processor 240 tothe display unit in the preview mode and, in the capture mode, parsesthe output of the post-processor 240 into the viewing images and stillimages such that the viewing image is output to the display unit 130 andthe capture image is output to the still image codec. The parser 260 canbe implemented in software or hardware. In the case of the softwareparser, the control unit 100 controls the viewing image buffer 230 andthe still image buffer 250 to select the image from the post-processor240 and controls the parser 260 to parse the viewing and still images atevery frame period. In the case of the hardware parser, a demultiplexer,such that the control unit 100 controls the demultiplexer to output theviewing and still images to the respective display unit 130 and thestill image codec 190, may be used.

The still image codec 190 encodes the capture image output from theparser 260 to be compressed, the compressed capture image being storedin the storage unit 120. Here, the still image codec 190 can be a JointPhotographic Experts Group (JPEG) codec or any other similar and/orsuitable codec for images. In the motion image recording mode, theviewing image parsed by the parser 260 is supplied to the motion imagecodec 180, as well as the display unit 130, such that the motion imagecodec 180 stores the viewing image in the storage unit 120 as acompressed image under the control of the control unit 100.

A description is made of the operation of the above-structured imageprocessor. If the user inputs a camera driving command to the input unit140, the control unit 100 drives the camera 110. The image taken by thecamera 110 can be a camera's full resolution Bayer image which is inputto the pre-processor 210. The pre-processor 210 generates a frame imageat every frame period under the control of the control unit 100. Here,the frame rate can be 30 Frames Per Second (fps) or higher, e.g., 60fps. The pre-processor 210 performs AWB, AE, AF extraction, lens shadingcorrection, dead pixel correction, knee correction, etc. on the frameimage. The pre-processed image is still a full resolution image andsupplied to the image scaler 220 and the buffer 250.

The image scaler 220 scales the full resolution image to fit the screensize of the display unit 130. Here, the image scaling can be performedwith at least one of resizing, decimation, interpolation, crop, andsumming and averaging. The image scaling reduces the number of pixels ofthe camera's full resolution image so as to fit the aspect ratio of thedisplay unit 130. Here, the image scaling can be performed at variousscaling ratios.

The image scaler 220 can be implemented with a resizer and/or a summingaverage unit. In the case that the image scaler 220 is implemented withthe resizer, the resizer resizes the full resolution image to be fit forthe display unit 130 in volume and size. At this time, resizing can beperformed with at least one of decimation, interpolation, and crop. Inthe case that the image scaler 220 is implemented with a summing averageunit, the summing average unit performs summing and averaging on theadjacent pixels of the image to reduce the number of pixels of theimage. The summing average unit averages the adjacent pixels in the fullresolution image output from the pre-processor 210 to reduce the numberof pixels of the viewing image. The summing average unit is capable ofenhancing a signal-to-noise ratio and the resolution throughpseudo-Foveon method. The pixels of the image output from the camera 110patterned such that four adjacent pixels consist of one R pixel, one Bpixel, and two G pixels. By averaging the four pixels into one pixel, itis possible to scale the image at the ratio of 4:1. In an exemplaryembodiment of the present invention, the description is made under theassumption that the summing average unit sums and averages the fourpixels into one pixel. In an exemplary embodiment of the presentinvention, the summing average unit sums and averages the two G pixelsinto one G pixel and processes the averaged G, R, and B pixels into onepixel. For example, pixels G11, B12, R21, and G22 are processed asR=R21, B=B12, and G=(G11+G22)/2 to be a pixel with an RGB value. Throughthe averaging operation, the four pixels are integrated into one pixelwith accompaniment of color interpolation. Through pixel averaging, itis possible to scale down the four pixels into one pixel. At this time,the pixels are color-interpolated. However, the present invention is notlimited thereto, and any suitable and/or similar number of pixels may becombined into one pixel.

The image-scaled pixels are buffered in the viewing image buffer 230.Here, the viewing image buffer 230 is responsible for adjusting theimage processing timing of the viewing images. In the capture mode, theviewing image buffer 230 buffers the viewing image of the next frameuntil the still image is processed completely.

The still image buffer 250 buffers the full resolution image output fromthe pre-processor 210 at every frame period. At this time, the stillimage buffer 250 can be configured to have a ring buffer structurecapable of buffering a predetermined number of frame images. Forexample, the still image buffer 250 can be implemented in the N-elementring buffer structure so as to buffer the frame image data output fromthe pre-processor 210 at every frame period. The frame image data arebuffered in the ring buffer from its first element and, if the buffer isfull, the oldest frame image data is overwritten with the new frameimage data. Here, N may be set to a value equal to or less than 5 and,in this case, it is preferred to configure the still image buffer 250 asa 5-element ring buffer capable of buffering 5 or less image data.Assuming that the shutter delay or the shutter lag is equal to 2 frames,the still image buffer 250 can be configured as a 3-element ring buffercapable of buffering 3 frame images, such that N=3.

The image acquired from the camera 110 at every frame is buffered in theviewing image buffer 230 as viewing images and in the still image buffer250 as still images. The control unit 100 controls the images bufferedin the viewing image buffer 230 and the still image buffer 250 to beoutput in a time divisional manner within a frame period.

A description is made of the preview mode and/or motion image recordingmode operations hereinafter.

As described above, the image taken by the camera at every frame ispre-processed and then scaled to be generated as a viewing image. Theviewing image buffer 230 buffers the viewing image, and the still imagebuffer 250 buffers the still image. The control unit 100 reads out thebuffered viewing image at the beginning part of the frame to thepost-processor 240. The post processor 240 performs color interpolation,IPC, and image conversion on the viewing image to generate a YUC image.In the case of using the summing average unit for image scaling, thecolor interpolator of the post-processor performs only the colorinterpolation function by skipping the full color conversion on thepixel data. The viewing image processed by the post-processor 240 issent from the parser 260 to the timing manager 170 and stored thereintemporarily. The control unit 100 reads out the viewing image storedtemporarily in the timing manager to the display unit and the storageunit 120 at a preset time of the frame period, e.g., the end part of theframe. The image processor according to an exemplary embodiment of thepresent invention processes the viewing image at the beginning part ofthe frame and sends the processed image to the display unit 130 and/orstorage unit 120 at a timing delayed as much as a predetermined durationfrom the frame period or at the end time of the frame. The reason forprocessing the viewing image at the end part of the frame or a partclose to the end of the frame is to process the still image in thecapture mode.

A description is made of the capture mode operation hereinafter.

If a user inputs a still image shooting or capture request to the inputunit 140 in the preview mode or motion image recording mode, then thecontrol unit 100 selects the frame image with zero shutter lag fromamong the frame images buffered in the still image buffer 250. Thecontrol unit 100 selects, in the capture mode, the viewing image storedin the viewing image buffer 230 and sends the selected image to thepost-processor 240 at the beginning part of the current frame period andcontrols the image output from the parser 260 to be sent to the timingmanager 170. The viewing image processed by the post-processor 240 isstored in the timing manager 170 temporarily through the parser 260.Afterward, the control unit 100 selects the zero shutter lag frame imagefrom the still image buffer 250 and sends the selected image to the postprocessor 240 at the residual part of the current frame. Thepost-processor 240 processes the input still image.

At this time, the still image can be the camera's full resolution imageor less than that in size. It takes more time to process the still imageof large size. In this case, the still image processing time may belonger than the frame period. Thus, if the time present in the currentframe period, i.e., at a time elapsed from the frame start time or atthe end time point, arrives in the middle of processing the still image,the control unit 100 sends the viewing image stored in the timingmanager 170 to the display unit 130 and motion image codec 180. If aviewing image of the next frame is generated in the middle of processingthe still image, the control unit 100 buffers the viewing image in theviewing image buffer 230. In this state, the post-processor 240 may beprocessing the still image of the previous frame. If the still image isprocessed completely, then the control unit 100 reads out the next frameviewing image buffered in the viewing image buffer 230 to thepost-processor 240 and controls the parser 260 to deliver the postprocessed image to the timing manager 260. The next frame viewing imageprocessed by the post processor 240 is sent to the timing manager 170through the parser 260, and the viewing image stored in the timingmanager 170 is read out to the display unit 130 and motion image codec180 at the preset time of the frame period under the control of thecontrol unit 100.

In the case that the still image is too large to be processed within oneframe period, the still image may be processed across two frames. Inthis case, the image processor processes the viewing image first in theframe period where the capture request was detected and the still imageat the residual part of the frame and at the beginning part of the nextframe continuously while buffering the viewing image of the next framein the viewing image buffer 230, and processes the buffered viewingimage upon completion of the still image processing. At this time, thetiming manager 170 stores the processed viewing image temporarily andoutputs the viewing image at a preset time of the frame period such thatthe viewing image is displayed on the screen of the display unit 130 atthe constant timing even when simultaneously processing the still image.Accordingly, when the capture command is input in the preview state ormotion image shooting state, the viewing image can be displayed at everyframe and even when the image processing time varies, the viewing imagecan be displayed at the same timing within the frame constantly.

Referring to FIG. 2B the image scaler 220 adjusts the full resolutionimage output from the camera 110 to the viewing image size that fits thedisplay unit 130 and/or the motion image size for storing as a motionimage. The viewing image buffer 230 buffers the viewing image outputfrom the image scaler 220. Here, the viewing image buffer 230 adjuststhe time for processing the viewing image within the frame period.

The still image buffer 250 buffers the full resolution image output fromthe camera 110. Here, the buffer 250 may be configured to be a sizecapable of buffering a number of frame images large enough to compensatefor the shutter lag of the camera device in a ring buffer structure. Thebuffer 250 buffers the full resolution image output from thepre-processor 210 at every frame and reads out a zero shutter lag imagefrom among the buffered images as a still image under the control of thecontrol unit 100.

The pre-processor 210 receives the viewing and/or still images outputfrom the viewing buffer 230 and/or the still image buffer 250 in a timedivisional manner and pre-processes the images. The control unit 100controls transfer of the viewing image buffered in the viewing imagebuffer 230 to the pre-processor 210 in the preview mode and/or motionimage recording mode and controls transfers of the images buffered inthe viewing image buffer 230 and the still image buffer 250 to thepre-processor 210 in a time divisional manner within the frame period inthe capture mode, e.g., snapshot mode. The pre-processor 210 is capableof performing AWB, AE, and AF extraction and processing, lens shadingcorrection, dead pixel correction, knee correction, etc. The postprocessor 240 performs color interpolation, noise reduction, colorcorrection, and image conversion to generate YUV data.

The parser 260 parses the images processed by the post-processor intothe viewing and/or still images. The parser 260 transfers the output ofthe post processor 240 to the display unit 130 in the preview mode andparses, in the capture mode, the output of the post-processor 240 intothe viewing image and the capture image, the viewing image beingtransferred to the display unit 130 and the capture image beingtransferred to the still image codec 260. The parser 260 can beimplemented as a software parser or a hardware parser. In the case ofthe software parser, the control unit 100 controls the viewing imagebuffer 230 and the still image buffer 250 to select the image to beoutput to the post-processor 240 at every frame period and controls theparser 260 to parse the images output from the post-processor 240 intothe viewing and still images. In the case of the hardware parser, it canbe implemented as a demultiplexer such that the control unit 100controls the demultiplexer to output the viewing image to the displayunit 130 and the still image to the still image codec 190.

The still image codec 190 encodes the capture image output from theparser 260 to be compressed, the compressed image being stored in thestorage unit 120. Here, the still image codec 190 can be a JPEG codec orany other similar and/or suitable codec for images. In the motion imagerecording mode, the viewing image parsed by the parser 260 issimultaneously supplied to the display unit 130 and the motion imagecodec 180 such that the motion image codec 180 encodes the viewing imagecompressively and stores the compressed motion image in the storage unit120 under the control of the control unit 100. The motion image codeccan be a H.264 codec or any other similar and/or suitable motion imagecodec.

In the image processor configured as shown in FIG. 2B, the output of thecamera 110 is supplied to the image scaler 220 and the still imagebuffer 250, and the pre-processor 210 processes the outputs of theviewing image buffer 230 and the still image buffer 250 in a timedivisional manner. Other configurations components of FIG. 2B areidentical with that of FIG. 2A, and the operations of the components arealso identical with those of the components of FIG. 2A.

Referring to FIG. 3A, the pre-processor 210 pre-processes the imageacquired from the camera 110. The image scaler 220 scales the camera'sfull resolution image output from the pre-processor 210 to the size thatwould fit the screen of the display unit 130 and/or the image size forstoring motion image. The viewing image buffer 230 buffers the viewingimage output by the image scaler 220. The still image buffer 250 buffersthe camera's full resolution image output from the pre-processor 210.Here, the buffer 250 can be configured to have a size large enough tobuffer the frame images capable of compensating the shutter lag of thecamera device in a ring buffer structure.

The multiplexer 270 multiplexes the images output from the viewing imagebuffer 230 and the still image buffer 250 under the control of thecontrol unit 100. The post-processor 240 performs color interpolation,noise reduction, color correction, and image conversion on the imagesmultiplexed by the multiplexer 270 to generate the YUC data. Thedemultiplexer 280 demultiplexes the output of the post-processor 240into the viewing and/or still images. The demultiplexer 280 supplies theoutput of the post-processor 240 to the display unit 130 in the previewmode and demultiplexes, in the capture mode, the output of the postprocessor 240 into the viewing and still images so as to output theviewing image to the display unit 130 and the capture image to the stillimage codec 260.

The image processor 200 configured as shown in FIG. 3A includes themultiplexer 270 and the demultiplexer 280 to establish the viewing imageand still image processing paths, and the other included components andtheir operations are identical with those of FIG. 2A. A description ismade of the image processor 200 configured as shown in FIG. 3Ahereinafter. In the preview mode, the control unit 100 sends a controlsignal to the multiplexer 270 to select the output of the image scaler220 and controls the demultiplexer 280 to supply the output of thepost-processor 240 to the display unit 130. In this case, thepost-processor 240 can be configured so as to process the image havingthe viewing image size.

In the preview mode, the image taken by the camera 110 is a fullresolution image, and the pre-processor 210 pre-processes the camera'sfull resolution image and outputs the processed image to the imagescaler 220 and the still image buffer 250 at every frame. The imagescaler 220 scales the full resolution image to a viewing image, and theviewing image buffer 230 buffers the viewing image. The still imagebuffer 250 buffers the camera's full resolution image output from thepre-processor up to the predetermined number. The multiplexer 270selects the output of the viewing image buffer 230 and the demultiplexer280 selects the output of the post-processor 240 under the control ofthe control unit 100. For example, the control unit 100 controls themultiplexer 270 and the demultiplexer 280 to establish the preview imageprocessing path but not the capture image processing path in the previewmode.

As described above, the image processor controls the multiplexer 270 andthe demultiplexer 280 to establish the viewing image processing path inthe preview mode or a motion image recording mode such that thepre-processor 210, the image scaler 220, the viewing image buffer 230,and the post-processor 240 operate to process the image taken by thecamera 110 into an image to be displayed on the display unit 130 atevery frame while the full resolution images are buffered in the stillimage buffer 250. At this time, the viewing image is in the down-scaledstate while the high resolution images are just being buffered forcapture, and the camera device is in the state of operating withrelatively low power consumption and heat generation.

Typically, a camera takes an image delayed a certain number of framesfrom the timing of the image presented on the viewfinder or display unit130 due to the shutter delay. For example, there is a delay, such as ashutter lag or time lag, between triggering the shutter and when thecamera records an image actually. The time delay is variable dependingon the camera, and the control unit 100 should know the delay timeacquired statistically. According to an exemplary embodiment of thepresent invention, the camera device buffers the images acquired by thecamera 110 by determining the shutter lag and selects the frame imagetaken at the timing when the shutter is triggered as a capture imagefrom among the buffered images, resulting in zero shutter lag.

If a capture request is input through the input unit 140 in the previewmode or motion image recording mode, then the control unit 100 detectsthe shutter press time and selects one of the images buffered in thestill image buffer 250, by determining the preset delay time, as thecapture image. The control unit 100 controls the multiplexer 270 and thedemultiplexer 280 to establish the viewing image and capture imageprocessing paths. For example, the control unit 100 controls themultiplexer 270 and the demultiplexer 280 to establish the viewing imageprocessing path and then the capture image processing pathssubsequently. Since the capture image or still image is high pixel data,it may not be processed within one frame period.

In the case that the still image is too large to be processed within oneframe period, the method according to an exemplary embodiment of thepresent invention processes the image across two frames. In this case,the control unit 100 controls the multiplexer 270 and the demultiplexer280 to establish the viewing image processing path first and then thecapture image processing path at the residual part of the frame period.The capture image processing path is maintained at the beginning part ofthe next frame such that the next frame image generated by the imagescaler 220 is buffered in the viewing image buffer 230. Once the stillimage processing has been completed, the control unit 100 controls themultiplexer 270 and the demultiplexer 280 to establish the viewing imageprocessing path such that the viewing image buffered in the viewingimage buffer 230 is output. The next frame viewing image is processedthrough the established viewing image processing path.

At this time, the viewing image output from the demultiplexer 280 isstored in the timing manager 170 temporarily and then output at thepreset time of the frame period. As aforementioned, the present part canbe positioned at the end of the frame or close to the end of the frame.Accordingly, even when the viewing image processing timing of the imageprocessor 200 varies, the viewing image output timing is adjusted by thetiming manager 170 such that the viewing images are displayed atconstant time position of the frame.

In the capture mode, the control unit 100 controls the multiplexer 270to select the viewing image buffered in the viewing image buffer 230 atthe beginning of the current frame period and to supply the selectedviewing image to the post-processor 240 and the demultiplexer 280 tooutput the processed viewing image to the timing manager 170.Accordingly, the viewing image buffered in the viewing image buffer 230is supplied to the post-processor 240 through the multiplexer 270 so asto be processed by the post-processor 240 and then transferred to thetiming manager 170 via the demultiplexer 280, the viewing image beingstored in the timing manager 170 temporarily.

Afterwards, the control unit 100 controls, at the residual part of theframe where the capture request has been occurred, the multiplexer 270to establish the capture image processing path in order to transfer thezero shutter lag frame image selected from the still image buffer 250 tothe post-processor 240 such that the post-processor 240 processes thezero shutter lag still image. At this time, the still image can be thecamera's full resolution image and, in this case, the still imageprocessing time may be longer than the frame period. In this case thecontrol unit 100 controls the multiplexer 270 and the demultiplexer 280to maintain the capture image processing path at the next frame. If thepreset time, which is a time point delayed as much as a predeterminedduration from the start point of the frame or end time point, arriveswithin the current frame in the middle of processing the still image,the control unit 100 controls to supply the viewing image temporarilystored in the timing manager 170 to the display unit 130 and motionimage codec 180.

If the next frame viewing image is generated in the middle of processingthe still image, then the control unit 100 controls buffering of theviewing image in the viewing image buffer 230. In this state, thepost-processor 240 may be in the state of processing the previous framestill image. If the still image processing has been completed, then thecontrol unit 100 controls the multiplexer 270 and the demultiplexer 280to establish the viewing image processing path, to read out the nextframe viewing image buffered in the viewing image buffer 230 to thepost-processor 240, and to supply the output of the post-processor 240to the timing manage 170. The next frame viewing image processed by thepost-processor 240 is transferred to the timing manager 170 via thedemultiplexer 280, and the viewing image stored in the timing manager170 is accessed and input to the display unit 130 and the motion imagecodec 180 at the preset time of the frame period under the control ofthe control unit 100.

Referring to FIG. 3B, the image scaler 220 adjusts the full resolutionimage output from the camera 110 to fit a screen size of the displayunit 130 and/or a size of the image to be stored as a motion image. Theviewing image buffer 230 buffers the viewing images output from theimage scaler 220. Here, the viewing image buffer 230 is responsible foradjusting the time of processing the viewing image within the frameperiod.

The still image buffer 250 buffers the full resolution image output fromthe camera 110. Here, the buffer 250 can be formed in a ring bufferstructure capable of buffering a predetermined number of frame images.The buffer 250 buffers the full resolution image output from thepre-processor 210 at every frame and, if a capture request is detected,outputs the zero shutter lag image selected from among the bufferedimage as a still image under the control of the control unit 100.

The multiplexer 270 multiplexes the outputs of the viewing image buffer230 and the still image buffer 250 under the control of the control unit100. The pre-processor 210 pre-processes the images multiplexed by themultiplexer 270. The post-processor 240 performs color interpolation,noise reduction, color correction, and image conversion on thepre-processed images to generate YUV data. The demultiplexer 280demultiplexes the image post-processed by the post-processor 240 intoviewing and/or still images. The demodulator 280 supplies the output ofthe post-processor 240 to the display unit 130 in the preview mode anddemultiplexes, in the capture mode, the output of the post-processor 240into the viewing and still images such that the viewing image issupplied to the motion image codec 180 and the capture image to thestill image codec 190.

The image processor configured as shown in FIG. 3B supplies the outputof the camera 110 to the image scaler 220 and the still image buffer 250simultaneously, and the pre-processor 210 is configured to process theoutputs of the viewing image buffer 230 and the still image buffer 250in a time divisional manner. The other components of the image processorof FIG. 3B are identical with those of FIG. 3A, and the operations ofthe individual components are also identical with those of the imageprocessor of FIG. 3A.

FIG. 4 is a block diagram illustrating an image processor processingmotion and still images according to an exemplary embodiment of thepresent invention. FIGS. 5 to 8 are timing diagrams illustratingoperation timings of an image processor, such as the image processor ofFIG. 4.

Referring to FIGS. 4 to 8, the image processor according to an exemplaryembodiment of the present invention stores a still image in the middleof processing motion image such that the viewing image is controlled tobe displayed in adaptation with the motion image stream. For example,the present exemplary embodiment proposes a method for displaying andstoring a motion image stably at every frame by controlling the imageprocessor processing the motion and still images in a time divisionalmanner to output the motion image at a preset time of the frame period.

Referring to FIG. 4, the image processor 200 is capable of be configuredas shown in any one of FIGS. 2A, 2B, 3A, and 3B. In FIG. 4, Time Monitor1 (TM1), Time Monitor 2 (TM2), and Time Controller (TC) denote timers.TM1 indicates the time when the motion image stream is input in theimage processor 200, TM2 indicates the time delay processed in the imageprocessor 200, and TC, which is for a memory read, indicates the timewhen the processed motion image is output finally. The timing manager170 can be implemented with a buffer, and FIG. 4 is depicted under theassumption that the timing manager 170 is implemented in a ring bufferstructure capable of storing three frames temporarily. A resizer 1 410and a resizer 2 420 are responsible for resizing an image to a motionimage recording size and viewing image size.

FIG. 5 is a timing diagram illustrating operation timings of an imageprocessor when a camera generates a motion image stream at a regularinterval according to an exemplary embodiment of the present invention.

The motion image stream generated by the camera 110 is input to theimage processor 200, as denoted by reference number 513, by a verticalsynchronization signal v-sync, as denoted by reference number 511. Atthis time, the video stream input to the image processor 200 at aregular interval, as denoted by reference number 513, is input to thetiming manager 170 with a delay as much as the image processing time“δ”, as denoted by reference number 517. In the case of the frame inwhich both the motion and still images have to be processed, such asframe D in FIG. 5, the image processor 200 may be incapable ofprocessing the still images. In this case, the image processor 200 mayprocess the video stream over a D frame period to a next frame E, havinga delay as much as “δ+α”. If the video stream processed in this way isdisplayed or compressed by the motion image codec 180 for playback, thenthe motion image is generated irregularly. In order to overcome thisproblem, the method, according to an exemplary embodiment of the presentinvention, records the image output by the camera 110, as denoted byreference number 513, with the V_sync, as denoted by reference number511, with the time TM1 in the image processor 200 and displays the imageoutput by the image processor 200 at the time TM2 synchronized withV_sync, as denoted by reference number 517. The video stream output bythe image processor 200, as denoted by reference number 517, is storedin the timing manager 170, i.e., the ring buffer, and is displayed onthe display unit 130 or encoded by the motion image codec 180 bydetermining the write and read times 41 and 42 in the timing manager170. For example, the video stream is regulated to be output at thepreset time of every frame period by controlling the read time of thevideo stream buffered in the timing manager 170.

At this time, TC is processed according to the following conditions:TC=TM1+Δ1, if a frame rate is regular;or, TC=TM2+Δ2+K,

wherein K=TM2−TM1=δ, if only a video stream is processed, and

K=δ−α, if a video stream and still image are processed together.

Accordingly, if a still image is stored in the middle of processing themotion image, which is frame D in FIG. 5, then the image processor 200receives the image from the camera 110, as denoted by reference number531, and generates a frame D viewing image, as denoted by referencenumber 541, and buffers the viewing image in the viewing image buffer230. The control unit 100 supplies the viewing image to the imageprocessor 200, which may include the post-processor in FIG. 2A andpre-processor and post-processor in FIG. 2B, to perform image processingoperation and buffers the processed image in the timing manager 170, asdenoted by reference number 551. The still image buffer 250 buffers thestill image, as denoted by reference number 531, and the control unit100 supplies the still image, i.e., zero shutter lag image, to the imageprocessor 200 in response to the shooting request. The image processor200 processes the still image as denoted by reference number 553. Atthis time, the image processor 200 occupies the time resource of frame Dperiod and frame E period to process the still image. Here, the E frameoccupancy time for still image processing is (δ+α).

In this case, the control unit 100 reads the viewing image buffered inthe timing manager 170 to the display unit 130 and motion image codec180 at the viewing image output time of frame D. In this case, the imageprocessor 200 is capable of displaying the viewing image of thecorresponding frame at the normal output timing in the middle ofprocessing the still image. The image generated in the frame E period,as denoted by reference number 535, can be generated by the imageprocessor 200 as the viewing image, as denoted by reference number 545,and buffered in the viewing image buffer 230. Once the still imageprocessing has completed, i.e., a time elapses as much as (δ+α) in theframe E period, then the control unit 100 detects this and supplies theprocessed still image to the still image codec 180 and reads out thebuffered viewing image of the frame E from the viewing image buffer 230at the timing 555 for processing. The processed viewing image isbuffered in the timing manager 170 at the timing 565, and the viewingimage buffered in the timing manager 170 is supplied to the display unit130 and motion image codec at the timing 565.

The image processor 200 for processing the motion and still images in atime-divisional manner, according to an exemplary embodiment of thepresent invention, processes the motion image first and stores theprocessed motion image temporarily and output the motion image at theend part of the frame. The image processor 200 capable of storing astill image while processing a motion image is capable of outputting themotion image streams at a regular interval regardless of the camera 110output interval.

FIG. 6 is a timing diagram illustrating operation timings of an imageprocessor when a camera generates a motion image stream at an irregularinterval according to an exemplary embodiment of the present invention.

In the case that the images output by the camera 110 are input to theimage processor 200 at a regular interval, as denoted by referencenumbers 611 and 613, the image processor 200 generates the viewing imageof the motion image stream, as denoted by reference number 615, and thevideo stream is processed by the image processor 200 with the delay asmuch as “δ”, as denoted by reference number 617. As described withreference to FIG. 5, when the still image is processed along with themotion image, the image processor 200 may undergo lack of capability andthus the motion image stream may be delayed as much as “δ+β” in the nextframe, wherein “δ+β” is a motion image processing time in frame E whenthe still image capture request is input in frame D. In this case, theimage processor controls processing of the viewing image 651, first, andthen the still image 653 at the residual part of the frame andcontinuously at the next frame, and then controls processing of the nextframe viewing image 655 at the residual part of the next frame. At thistime, the viewing image output timing is controlled to occur at the endpart of every frame period. In this case, the image processor is capableof processing the images output by the camera to be displayed or encodedby the codec at a regular interval on the same conditions as describedwith reference to FIG. 5.TC=TM1+Δ1, if frame rate is regular;or TC=TM2+Δ2+K,

wherein, K=TM2−TM1=δ, if only a video stream is processed, and

K=δ−β, if a video stream and still image are processed together.

FIG. 7 is a timing diagram illustrating operation timings of an imageprocessor when a video stream is input at an irregular interval and areference time is a frame end according to an exemplary embodiment ofthe present invention. FIG. 7 shows an exemplary case where the videostream is processed at the end part of the frame and other operationsare identical with those of FIG. 6.

FIG. 8 is a timing diagram illustrating operation timings of an imageprocessor when a reference time is set to frame end according to anexemplary embodiment of the present invention.

Referring to FIG. 8, when the synchronization signal is generated, asdenoted by reference number 811, the camera 110 outputs images, asdenoted by reference number 813. In the case that the reference time isset to the frame end, the preview images are output after δ from theinterrupt occurrence time Ai at the raw frame end, as denoted byreference number 813. If a still image shooting request is input in theframe A4, the preview image is output after δ in the frame A4, and thereis a need of minimum V_blank time at least for image processing, thenthe minimum V_blank time varies depending on the image size. In the caseof processing a still image 853 immediately after processing a previewimage, the still image processing is initiated at the time point afterthe time duration of β elapses. The still image processing continues tothe next frame, which is frame A5, and, the time delay between thecompletion of the still image processing and the output timing of thenext frame preview image, viewing image of frame A5, is φ. At this time,β has to be greater than the minimum V_blank time of the preview image,and φ has to be greater than the minimum V_blank time of the stillimage.

FIGS. 4 to 8 show the exemplary cases of image processors and outputtimings of the motion and still images in the image processors operatingin a time divisional manner. However, the camera device orcamera-equipped electronic device may be provided with an imageprocessor of FIG. 4 and an extra application processor. In this case,the image processor processes the images taken by the camera 110 whilethe application processor performs displaying images processed by theimage processor and storing the motion and still images. In this case,the camera device can be configured to operate in such a way that theimage processor, which processes the still and motion images in a timedivisional manner, processes δ2 and α as metadata of each frame and theapplication processor controls the output of the motion and still imagesfinally.

FIG. 9 is a flowchart illustrating an image processing method of acamera device or camera-equipped electronic device according to anexemplary embodiment of the present invention.

Referring to FIG. 9, the camera devices can be configured as shown inany of FIGS. 1 and 4, and the image processor can be configured as shownin any of FIGS. 2A, 2B, 3A, and 3C. The motion image processing modeincludes both the preview mode and the motion image recording mode. Asshown in FIG. 9, if it is determined at step 911 that a camera drivingcommand is input through the input unit 140, then the control unit 100drives the camera 110 at step 913 and determines if it has received theframe image from the camera 110 at step 913. Next, the control unit 100determines whether the camera is operating in the capture mode at step915 and, if the camera is not in the capture mode, process the motionimage so as to read out the processed viewing image in a preset time ofthe frame period at step 917. For example, when the device operates inthe motion image processing mode, the control unit 100 processes theframe images output from the camera 110 to output viewing images at thepreset time of the frame period. After the completion of the motionimage processing, the control unit 100 determines whether a proceduretermination command, such as an END command, is detected at step 919and, if so, ends the procedure and, otherwise, returns the procedure tostep 913. If the camera is operating in the capture mode in which themotion and still images are processed simultaneously, then the controlunit 100 detects the capture mode operation at step 915 and controls theimage processor to process the motion and still images in a timedivisional manner at step 921.

FIG. 10 is a flowchart illustrating a viewing image processing procedurein a motion image processing mode of an image processor according to anexemplary embodiment of the present invention.

Referring to FIG. 10, if a frame image is received from the camera 110,then the control unit 100 scales the frame image to a viewing image andbuffers the viewing image in the viewing image buffer 230 whilebuffering the camera's full resolution image in the still image buffer250 at step 1011. Next, the control unit 100 establishes a viewing imageprocessing path and processes the viewing image at step 1015. The imageprocessing may include the pre-processing and post-processing steps oronly the post-processing step. Next, the control unit 100 controls thetiming manager 170 to buffer the processed viewing image temporarily, atstep 1017, and, if a preset time arrives within the corresponding frame,then detects this at step 1019 and outputs the temporarily storedviewing image to the display unit 130 and the motion image codec 180 atstep 1021.

FIG. 11 is a flowchart illustrating a procedure of processing motion andstill images in an image processing method according to an exemplaryembodiment of the present invention.

Referring to FIG. 11, the control unit 100 controls scaling of the imageoutput from the camera 110 to generate a viewing image and buffering ofthe viewing image in the viewing image buffer 230 at step 1111. At thistime, the control unit 100 also controls the images output from thecamera 110 to be buffered in the still image buffer 250. Afterward, thecontrol unit 100, at step 1115, controls selection and processing of oneof the viewing images buffered in the viewing image buffer 230 andstores or buffers the processed viewing image, at step 1117, in thetiming manager 170 temporarily. The reason for storing the viewing imagein the timing manager 170 temporarily is to regulate the viewing imageoutput timing even when processing the still image.

After the viewing image is stored in the timing manager 170, the controlunit 100 selects the zero shutter lag still image from among the stillimages buffered in the still image buffer 250 at step 1119. At thistime, the still image, which is a high pixel image, can be processed forduration longer than the frame period. If the viewing image outputtiming, which is a preset time, arrives in the middle of processing thestill image, i.e., the time assigned for viewing image within the framearrives, then the control unit 100 detects this at step 1121, controlsoutputting of the viewing image stored in the timing manager 170 to thedisplay unit 130 and the motion image codec 180 at step 1123, andreturns the procedure to step 1119 to continue the still imageprocessing procedure.

If the next frame arrives in the middle of processing the still image atstep 1119, then the control unit 100 detects receipt of the next frameimage from the camera 110 at step 1125 and scales the image to a viewingimage and buffers the scaled viewing image in the viewing image buffer230 at step 1127 while buffering the received image in the still imagebuffer 250. The control unit 100 continues processing the still imagewhile receiving and buffering the next frame image.

If the still image processing has been completed while in this state,then the control unit 100 detects this at step 1129, outputs the stillimage to the still image codec 190 and processes the viewing imagebuffered in the viewing image buffer 230 at step 1131, and stores, orbuffers, the processed viewing image in the timing manager 170temporarily at step 1133. If the viewing image output time, which is thepreset time, arrives, then the control unit 100 detects this at step1135 and reads out the temporarily stored viewing image to the displayunit 130 and the motion image codec 180 at step 1137.

As described above, the image processor 200, according to an exemplaryembodiment of the present invention, processes the motion and stillimages in a time divisional manner and, when a still image capturecommand is input in the motion image processing mode, rearranges theoutput timing of the viewing image such that the viewing image isdisplayed on the display 130 at a constant timing in the frame period.

FIG. 12 is a flowchart illustrating a procedure of controlling viewingimage output timing of an image processor according to an exemplaryembodiment of the present invention.

Referring to FIG. 12, the control unit 100 monitors arrival of a framestart/end time of the image acquired from the camera 110 and outputs theimage to the image processor at step 1211. The control unit 100 monitorsthe frame start/end time of the image processor output, which is outputfrom the camera, at step 1213. For example, the image processor 200monitors the start and end times of the image output from the camera 110and the input and processing times of the image and then, at step 1215,buffers the viewing images processed by the image processor 200 in thetiming manager 170. Next, the control unit 100 outputs the viewing imagestored temporarily in the timing manager 170 with the delay as long as(δ+α) at step 1217. Here, δ denotes the image processing time of theimage processor 200, and α denotes the still image processing timecontinued at the next frame in processing the still image. By delayingthe viewing image output time as much as (δ+α) at every frame, the imageprocessor is capable of outputting the viewing images at a constantposition within the frame even when processing the still image together.The control unit 100 delays the output of the viewing image as much as(δ+α) by means of the timing manager 170, at step 1217, and displays theviewing image on the screen of the display unit 130 at step 1219,thereby maintaining the viewing image output timing regularly.

FIG. 13 is a flowchart illustrating a procedure of outputting a viewingimage in processing motion image in a motion image processor accordingto an exemplary embodiment of the present invention. FIG. 14 is aflowchart illustrating a procedure of outputting a viewing image when animage processor processes motion and still images in a time divisionalmanner according to an exemplary embodiment of the present invention. Adescription is made of the procedures of FIGS. 13 and 14 with referenceto FIGS. 1, 2A, and 5.

Referring to FIG. 13, if the camera 110 captures and outputs an image,the pre-processor 210 of the image processor 200 receives andpre-processes the image at step 1311. Next, the image scaler 220 scalesthe pre-processed image to be the viewing images, as denoted byreference number 515 of FIG. 5, at step 1313. Next, the viewing imagebuffer 230 buffers the viewing images at step 1315. The still imagebuffer 250 buffers the camera's full resolution image at step 1323. Thecontrol unit 100 accesses the viewing image buffered in the viewingimage buffer 230 and reads out the accessed viewing images to thepost-processor 240, and the post-processor 240 post-processes theviewing images output from the viewing image buffer 230 at step 1317. Inthe case of the image processor 200, configured as shown in FIG. 2B, theimage buffered in the viewing image buffer 230 is transferred to thepre-processor 210 and then the post-processed viewing image istransferred to the post-processor 240 so as to be post-processed. Here,the image processing time can be δ. The processed image is input to thetiming manager 170 such that the timing manager 170 stores the viewingimage temporarily at step 1319 and outputs the viewing image to thedisplay unit 130 when the preset time of the frame period arrives atstep 1321. Here, the preset time can be the frame end position or aposition close to the frame end and can be (Δ+δ), as shown in FIG. 5,where Δ can be the value including the still image processing time inprocessing both the motion and still images time-divisionally. It ispreferred that (Δ+δ) is configured at the frame end or a position closeto the frame end.

Referring to FIG. 14, if the camera 110 takes and outputs images in thecapture mode, the pre-processor 210 of the image processor 200pre-processes the image at step 1411. Next, the image scaler 220 scalesthe image to generate the viewing image, as denoted by reference number515 of part 551 in FIG. 5, at step 1413, and then, at step 1415, theviewing image buffer 230 buffers the viewing image. The still imagebuffer 250 buffers the camera's full resolution image at step 1451. Thecontrol unit 100 accesses the viewing image buffered in the viewingimage buffer 230 and reads out the accessed viewing image to thepost-processor 240, and the post processor 240 post-processes theviewing image at step 1417. Here, the image processor 200, configured asshown in FIG. 2B, transfers the image buffered in the viewing imagebuffer 230 to the pre-processor 210 and then transfers the pre-processedviewing image to the post-processor 240. The post-processed viewingimage is input to the timing manager 170, and the timing manager 170stores the viewing image temporarily at step 1419.

After the viewing image is stored in the timing manager 170, the controlunit 100 accesses the still image with zero shutter lag in the stillimage buffer 250 and reads out the zero shutter lag image to the postprocessor 240. For example, the control unit 100 processes the imagesoutput the viewing image buffer 230 and the still image buffer 250 in atime divisional manner within the frame D period such that the stillimage is transferred to the post-processor 240 and the post-process ofthe viewing image of frame D. The post-processor 240 post-processes thestill image at step 1453 and then outputs the still image at step 1455.

When post-processing the still image of frame D, if the viewing imageoutput timing arrives while processing the still image, as denoted byreference number 553 of FIG. 5, then the control unit 100 controls thetiming manager 170 to output the temporarily stored viewing image, at apreset time, to the display unit 130 at step 1421. At this time, thepost-processor 240 is processing the still image continuously, asdenoted by reference number 553 of FIG. 5.

If the next frame, i.e., frame E, image arrives from the camera 110 inthe above state, then the pre-processor 210 receives the next frameimage at step 1423, then the image scaler 220 scales the received imageto the viewing image, and the viewing image buffer 230 buffers theviewing image at step 1425. Even at this time, the post-processor 240may be processing the still image, as denoted by reference number 553 ofFIG. 5.

If the post-processing of the still image has been completed, then thecontrol unit 100 detects this and parses and transfers the output of thepost-processor 240 to the still image codec 190. The control unit 100reads out the viewing image buffered in the viewing image buffer 230 tothe post-processor 240. The post-processor 240 post-processes theviewing image at step 1427 and, if the post-processing has completed,then parses and transfers the output of the post-processor 240 to thetiming manager 170. The timing manager 170 stores the processed viewingimage temporarily at step 1429 and, when a preset time arrives, outputsthe temporarily stored viewing image to the display unit 130 at step1431.

At this time, the preset time may occur at the same position or timewithin the frames D and E. For example, the preset time can be the frameend position or a position close to the frame end, and can be set to(Δ+δ), as shown in FIG. 5, where Δ can be the value including the stillimage processing time in processing both the motion and still imagestime-divisionally. It is preferred that (Δ+δ) is configured at the frameend or a position close to the frame end.

As described above, the image processor of the camera device orcamera-equipped electronic device according to the present exemplaryembodiments includes a high pixel camera and processes images in a timedivisional manner, whereby the image processor being capable ofoutputting the motion viewing image at a constant time interval to thedisplay unit while processing the still image to be stored in the motionimage processing mode.

While the invention has been shown and described with reference tocertain exemplary embodiments thereof, it would be understood by thoseskilled in the art that various changes in form and detail can be madetherein without departing from the spirit and scope of the invention asdefined by the appended claims and their equivalents.

What is claimed is:
 1. An image processing apparatus comprising: animage processor configured to process for separating a video stream intoan motion image and a still image, to buffer a predetermined number offrames of the motion image, and to buffer a predetermined number offrames of the still image, the video stream generated by a camera atevery frame period; a timing manager configured to store the motionimage processed by the image processor temporarily and to output themotion image at a preset time of a frame period; and in response to acapture request time point, a controller configured to control the imageprocessor: to process a current frame of the motion image in response tothe capture request and to output the current frame of the motion imageon a display, to process one of the buffered frames of the still imagein response to the capture request, when the processing of the currentframe of the motion image is completed, and to process to a next frameof the motion image, wherein the controller is further configured tocontrol the timing manager to store the next frame of the motion imageprocessed by the image processor upon completion of processing one ofthe buffered frames of the still image.
 2. The apparatus of claim 1,wherein the image processor comprises: an image scaler configured toscale the video stream generated by a camera into; a motion image bufferconfigured to buffer the motion image; and a still image bufferconfigured to buffer the image, the image processor processing themotion image and the still image at given time periods.
 3. The apparatusof claim 2, wherein the image processor further comprises: apre-processor configured to pre-process an image of the video streamfrom the camera and to transfer the pre-processed image of the videosteam to the image scaler; a post-processor configured to post-processthe motion image and the still image that are input at given timeperiods; and a parser configured to parse the post-processed motionimage and the post-processed still image.
 4. The apparatus of claim 3,wherein the controller is further configured to control the motion imagebuffer to output the buffered motion image to the post-processor, whichpost-processes the motion image, and the timing manager, to store thepost-processed motion image temporarily, and to output the motion imageat the preset time of the frame period in a motion image processingmode.
 5. The apparatus of claim 4, wherein the controller is furtherconfigured to control post-processing of the motion image buffered inthe motion image buffer and storing of the post-processed motion imagein the timing manager temporarily in the current frame period, whereinthe controller is further configured to control post-processing of thestill image stored in the still image buffer that stores the residualpart of the current frame period to the beginning part the next frameperiod, wherein the controller is further configured to controlprocessing of the motion image of the next frame period which isbuffered in the motion image buffer upon completion of processing thestill image, and wherein the controller is further configured to controloutputting of the motion image stored in the timing manager at thepreset time of every frame period.
 6. The apparatus of claim 4, whereinthe preset time of the frame period comprises an end point of the frameperiod of the motion image.
 7. The apparatus of claim 2, wherein theimage processor further comprises: a pre-processor configured topre-process the motion image and the still image input in at given timeperiods; a post-processor configured to post-process the pre-processedimages; and a parser configured to parse the post-processed motion imageand the post-processed still image.
 8. The apparatus of claim 7, whereinthe controller is further configured to control transferring of themotion image that is buffered in the motion image buffer to thepost-processor, wherein the controller is further configured to controlpost-processing of the motion image, wherein the controller is furtherconfigured to control temporarily storing of the post-processed motionimage in the timing manager, and wherein the controller is furtherconfigured to control outputting of the motion image stored in thetiming manager at the preset time of frame period in a motion imageprocessing mode.
 9. The apparatus of claim 8, wherein the controller isfurther configured to control post-processing of the image that isbuffered in the motion image buffer and storing of the post-processedmotion image in the timing manager in current frame period, wherein thecontroller is further configured to control post-processing of the stillimage buffered in the still image buffer at a residual part of thecurrent frame period to the beginning part of the next frame periodcontinuously, wherein the controller is further configured to controlprocessing of the motion image of the next frame period which isbuffered in the motion image buffer upon completion of processing thestill image, and wherein the controller is further configured to controloutputting of the motion image stored in the timing manager at thepreset time of every frame period.
 10. The apparatus of claim 9, whereinthe preset time of the frame period comprises an end point of the frameperiod of the motion image.
 11. An image processing method comprising:processing for separating a video stream into an motion image and anstill image, buffering a predetermined number of frames of the motionimage, and buffering a predetermined number of frames of the stillimage, wherein the video stream generated by a camera at every frameperiod; storing the processed motion image temporarily; outputting thetemporarily stored motion image at a preset time of a frame period;processing a current frame of the motion image in response to thecapture request, and outputting the current frame of the motion image ona display; processing one of the buffered frames of the still image inresponse to the capture request, when the processing of the currentframe of the motion image is completed; processing to a next frame ofthe motion image; and storing the processed next frame of the motionimage upon completion of processing one of the buffered frames of thestill image.
 12. The method of claim 11, wherein the processing of theimage generated by the camera at every frame period comprises:pre-processing an image of the video stream; scaling the pre-processedimage to the motion image; post-processing the motion image and thestill image; and parsing the post-processed motion image and thepost-processed still image.
 13. The method of claim 11, wherein theprocessing of the image generated by the camera at every frame periodcomprises: scaling an image of the video stream to the motion image;pre-processing the motion image and the still image; post-processing thepre-processed motion image and the pre-processed still image; andparsing the post-processed motion image and the post-processed stillimage.